<!DOCTYPE html>
<html>
<head>
  <!-- hexo-inject:begin --><!-- hexo-inject:end --><meta charset="utf-8">
  
  <title>CreateMoMo</title>
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
  <meta property="og:type" content="website">
<meta property="og:title" content="CreateMoMo">
<meta property="og:url" content="http://createmomo.github.io/index.html">
<meta property="og:site_name" content="CreateMoMo">
<meta property="og:locale" content="default">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="CreateMoMo">
  
  
    <link rel="icon" href="/favicon.png">
  
  
    <link href="//fonts.googleapis.com/css?family=Source+Code+Pro" rel="stylesheet" type="text/css">
  
  <link rel="stylesheet" href="/css/style.css"><!-- hexo-inject:begin --><!-- hexo-inject:end -->
  

</head>

<body>
  <!-- hexo-inject:begin --><!-- hexo-inject:end --><div id="container">
    <div id="wrap">
      <header id="header">
  <div id="banner"></div>
  <div id="header-outer" class="outer">
    <div id="header-title" class="inner">
      <h1 id="logo-wrap">
        <a href="/" id="logo">CreateMoMo</a>
      </h1>
      
    </div>
    <div id="header-inner" class="inner">
      <nav id="main-nav">
        <a id="main-nav-toggle" class="nav-icon"></a>
        
          <a class="main-nav-link" href="/">Home</a>
        
          <a class="main-nav-link" href="/archives">Archives</a>
        
      </nav>
      <nav id="sub-nav">
        
        <a id="nav-search-btn" class="nav-icon" title="Search"></a>
      </nav>
      <div id="search-form-wrap">
        <form action="//google.com/search" method="get" accept-charset="UTF-8" class="search-form"><input type="search" name="q" class="search-form-input" placeholder="Search"><button type="submit" class="search-form-submit">&#xF002;</button><input type="hidden" name="sitesearch" value="http://createmomo.github.io"></form>
      </div>
    </div>
  </div>
</header>
      <div class="outer">
        <section id="main">
  
    <article id="post-Table-of-Contents" class="article article-type-post" itemscope itemprop="blogPost">
  <div class="article-meta">
    <a href="/2019/07/18/Table-of-Contents/" class="article-date">
  <time datetime="2019-07-18T00:08:28.000Z" itemprop="datePublished">2019-07-18</time>
</a>
    
  </div>
  <div class="article-inner">
    
    
      <header class="article-header">
        
  
    <h1 itemprop="name">
      <a class="article-title" href="/2019/07/18/Table-of-Contents/">Table of Contents</a>
    </h1>
  

      </header>
    
    <div class="article-entry" itemprop="articleBody">
      
        <hr>
<h4 id="TECHNICAL-ARTICLES"><a href="#TECHNICAL-ARTICLES" class="headerlink" title="TECHNICAL ARTICLES"></a><strong><em>TECHNICAL ARTICLES</em></strong></h4><p><strong><em>Quick Links:</em></strong></p>
<ul>
<li><a href="https://github.com/createmomo" target="_blank" rel="external">[Github] https://github.com/createmomo</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#tableofcontents">[2019] Probabilistic Graphical Models Revision Notes</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#tableofcontents">[2018] Super Machine Learning Revision Notes</a></li>
<li>[2017] CRF Layer on the Top of BiLSTM<ul>
<li><a href="https://createmomo.github.io/2017/09/12/CRF_Layer_on_the_Top_of_BiLSTM_1/">CRF Layer on the Top of BiLSTM - 1</a> Outline and Introduction</li>
<li><a href="https://createmomo.github.io/2017/09/23/CRF_Layer_on_the_Top_of_BiLSTM_2/">CRF Layer on the Top of BiLSTM - 2</a> CRF Layer (Emission and Transition Score)</li>
<li><a href="https://createmomo.github.io/2017/10/08/CRF-Layer-on-the-Top-of-BiLSTM-3/">CRF Layer on the Top of BiLSTM - 3</a> CRF Loss Function</li>
<li><a href="https://createmomo.github.io/2017/10/17/CRF-Layer-on-the-Top-of-BiLSTM-4/">CRF Layer on the Top of BiLSTM - 4</a> Real Path Score</li>
<li><a href="https://createmomo.github.io/2017/11/11/CRF-Layer-on-the-Top-of-BiLSTM-5/">CRF Layer on the Top of BiLSTM - 5</a> The Total Score of All the Paths</li>
<li><a href="https://createmomo.github.io/2017/11/24/CRF-Layer-on-the-Top-of-BiLSTM-6/">CRF Layer on the Top of BiLSTM - 6</a> Infer the Labels for a New Sentence</li>
<li><a href="https://createmomo.github.io/2017/12/06/CRF-Layer-on-the-Top-of-BiLSTM-7/">CRF Layer on the Top of BiLSTM - 7</a> Chainer Implementation Warm Up</li>
<li><a href="https://createmomo.github.io/2017/12/07/CRF-Layer-on-the-Top-of-BiLSTM-8/">CRF Layer on the Top of BiLSTM - 8</a> Demo Code<br><img src="/2019/07/18/Table-of-Contents/search.png" alt="Searching..."></li>
</ul>
</li>
</ul>
<p><strong><em>Detailed Links:</em></strong><br><strong>(Last Updated: 2019.09.15) <a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#tableofcontents">Probabilistic Graphical Models Revision Notes</a></strong></p>
<ul>
<li><strong><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#representations">Representations</a></strong><ul>
<li><strong><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#bayesian_network">Bayesian Network (directed graph)</a></strong><ul>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#defination">Defination</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#reasoning_patterns_in_bayesian_network">Reasoning Patterns in Bayesian Network</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#flow_of_probabilistic_influence">Flow of Probabilistic Influence (active trial)</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#independencies">Independencies</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#d_seperation">d-seperation</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#i_maps">I-Maps (Indenpendency Map)</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#factorisation_and_i_maps">Factorisation and I-Maps</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#naive_bayes">Naive Bayes</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#template_models">Template Models</a><ul>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#temporal_models">Temporal Models (involve over time)</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#2tbn">2 Time-Slice Bayesian Network (2TBN)</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#plate_models">Plate Models</a></li>
</ul>
</li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#conditional_probability_distribution">Conditional Probability Distribution (CPD)</a><ul>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#general_cpd">General CPD</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#table_based_cpd">Table-based CPD</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#context_specific_independence">Context-specific Independence</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#tree_structured_cpd">Tree-Structured CPD</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#multiplexer_cpd">Multiplexer CPD</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#noise_or_cpd">Noise OR CPD</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#sigmoid_cpd">Sigmoid CPD</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#continuous_variables">Continuous Variables</a></li>
</ul>
</li>
</ul>
</li>
<li><strong><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#markov_network">Markov Network (undirected graph)</a></strong><ul>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#markov_network_fundamentals">Markov Network Fundamentals</a><ul>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#pairwise_markov_networks">Pairwise Markov Networks</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#general_gibbs_distribution">General Gibbs Distribution (a more general expression)</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#induced_markov_network">Induced Markov Network (connects every pair of nodes that are in the same factor)</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#factorization">Factorization</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#conditional_random_fields">Conditional Random Fields</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#independencies_in_markov_networks">Independencies in Markov Networks</a></li>
</ul>
</li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#local_structure_in_markov_networks">Local Structure in Markov Networks</a><ul>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#log_linear_models">Log-linear Models (CRF, Ising Model, Metric MRFs)</a></li>
</ul>
</li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#decision_making">Decision Making</a><ul>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#maxium_expected_utility">Maximum Expected Utility</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#utility_functions">Utility Functions</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#value_of_perfect_information">Value of Perfect Information</a></li>
</ul>
</li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#knowledge_engineering">Knowledge Engineering</a><ul>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#generative_vs_descriminative">Generative vs. Discriminative</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#designing_a_graphical_model">Designing a graphical model (variable types)</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#structure">Structure</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#parameters_local_structure">Parameters: Local Structure</a></li>
<li><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#iterative_refinement">Iterative Refinement</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
<li><strong><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#inference">Inference</a></strong></li>
<li><strong><a href="https://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#learning">Learning</a></strong><br><img src="/2019/07/18/Table-of-Contents/dog-pgm-mini.png" alt="The Power of Probabilistic Graphical Models"></li>
</ul>
<p><strong>(Last Updated: 2019.01.06) <a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#tableofcontents">Super Machine Learning Revision Notes</a></strong></p>
<ul>
<li><strong><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#activation_functions">Activation Functions</a></strong></li>
<li><strong><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#gradient_descent">Gradient Descent</a></strong><ul>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#computation_graph">Computation Graph</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#backpropagation">Backpropagation</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#gradients_for_l2_regularization">Gradients for L2 Regularization (weight decay)</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#vanishing_exploding_gradients">Vanishing/Exploding Gradients</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#mini_batch_gradient_descent">Mini-Batch Gradient Descent</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#stochastic_gradient_descent">Stochastic Gradient Descent</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#choosing_mini_batch_size">Choosing Mini-Batch Size</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#gradient_descent_with_momentum">Gradient Descent with Momentum (always faster than SGD)</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#gradient_descent_with_rmsprop">Gradient Descent with RMSprop</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#adam">Adam (put Momentum and RMSprop together)</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#learning_rate_decay_methods">Learning Rate Decay Methods</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#batch_normalization">Batch Normalization</a></li>
</ul>
</li>
<li><strong><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#parameters">Parameters</a></strong><ul>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#learnable_and_hyper_parameters">Learnable and Hyper Parameters</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#parameters_initialization">Parameters Initialization</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#hyper_parameter_tuning">Hyper Parameter Tuning</a></li>
</ul>
</li>
<li><strong><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#regularization">Regularization</a></strong><ul>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#l2_regularization">L2 Regularization (weight decay)</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#l1_regularization">L1 Regularization</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#dropout">Dropout (inverted dropout)</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#early_stopping">Early Stopping</a></li>
</ul>
</li>
<li><strong><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#models">Models</a></strong><ul>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#logistic-regression">Logistic Regression</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#multiclass_classification">Multi-Class Classification (Softmax Regression)</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#transfer_learning">Transfer Learning</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#multitask_learning">Multi-task Learning</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#convolutional_neural_network">Convolutional Neural Network (CNN)</a><ul>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#filter_kernel">Filter/Kernel</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#stride">Stride</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#padding">Padding (valid and same convolutions)</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#a_convolutional_layer">A Convolutional Layer</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#1_1_convolution">1*1 Convolution</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#pooling_layer">Pooling Layer (Max and Average Pooling)</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#lenet_5">LeNet-5</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#alexnet">AlexNet</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#vgg_16">VGG-16 </a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#resnet">ResNet (More Advanced and Powerful) </a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#inception_network">Inception Network </a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#object_detection">Object Detection </a><ul>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#classification_with_localisation">Classification with Localisation</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#landmark_detection">Landmark Detection</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#sliding_windows_detection_algorithm">Sliding Windows Detection Algorithm</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#region_proposal">Region Proposal (R-CNN)</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#yolo_algorithm">YOLO Algorithm</a><ul>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#bounding_box_predictions">Bounding Box Predictions (Basics of YOLO)</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#intersection_over_union">Intersection Over Union</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#non_max_suppression">Non-max Suppression</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#anchor_boxes">Anchor Boxes</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#face_verification">Face Verification</a><ul>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#one_shot_learning">One-Shot Learning (Learning a “similarity” function)</a><ul>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#siamese_network">Siamese Network</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#triplet_loss">Triplet Loss</a></li>
</ul>
</li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#face_recognition_verification_and_binary_classification">Face Recognition/Verification and Binary Classification</a></li>
</ul>
</li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#neural_style_transfer">Neural Style Transfer </a></li>
</ul>
</li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#sequence_models">Sequence Models</a><ul>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#recurrent_neural_network">Recurrent Neural Network Model</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#gated_recurrent_unit">Gated Recurrent Unit (GRU)</a><ul>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#gru_simplified">GRU (Simplified)</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#gru_full">GRU (Full)</a></li>
</ul>
</li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#long_short_term_memory">Long Short Term Memory (LSTM)</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#bidirectional_rnn">Bidirectional RNN</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#deep_rnn_example">Deep RNN Example</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#word_embedding">Word Embedding</a><ul>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#one_hot">One-Hot</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#embedding_matrix">Embedding Matrix ($E$)</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#learning_word_embedding">Learning Word Embedding</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#word2vec_and_skip_gram">Word2Vec &amp; Skip-gram</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#negative_sampling">Negative Sampling</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#glove_vector">GloVe Vector</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#elmo">Deep Contextualized Word Representations (ELMo, Embeddings from Language Models)</a></li>
</ul>
</li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#sequence_to_sequence_model_example">Sequence to Sequence Model Example: Translation</a><ul>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#pick_the_most_likely_sentence">Pick the most likely sentence (Beam Search)</a><ul>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#beam_search">Beam Search</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#length_normalisation">Length Normalisation</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#error_analysis_in_beam_search">Error Analysis in Beam Search (heuristic search algorithm)</a></li>
</ul>
</li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#bleu_score">Bleu Score</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#combined_bleu">Combined Bleu</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#attention_model">Attention Model</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#transformer">Transformer (Attention Is All You Need)</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#bert">Bidirectional Encoder Representations from Transformers (BERT)</a></li>
</ul>
</li>
<li><strong><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#tips">Practical Tips</a></strong><ul>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#train_dev_test">Train/Dev/Test Dataset</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#over_and_under_fitting">Over/UnderFitting, Bias/Variance, Comparing to Human-Level Performance, Solutions</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#mismatched_data_distribution">Mismatched Data Distribution</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#input_normalization">Input Normalization</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#single_number_model_evaluation_metric">Use a Single Number Model Evaluation Metric</a></li>
<li><a href="https://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#error_analysis">Error Analysis (Prioritize Next Steps)</a></li>
</ul>
</li>
</ul>
<p><img src="/2019/07/18/Table-of-Contents/dog-dl.png" alt="Reviewing..."></p>
<p><strong>[2017] CRF Layer on the Top of BiLSTM (BiLSTM-CRF)</strong></p>
<ul>
<li><a href="https://createmomo.github.io/2017/09/12/CRF_Layer_on_the_Top_of_BiLSTM_1/">CRF Layer on the Top of BiLSTM - 1</a> Outline and Introduction</li>
<li><a href="https://createmomo.github.io/2017/09/23/CRF_Layer_on_the_Top_of_BiLSTM_2/">CRF Layer on the Top of BiLSTM - 2</a> CRF Layer (Emission and Transition Score)</li>
<li><a href="https://createmomo.github.io/2017/10/08/CRF-Layer-on-the-Top-of-BiLSTM-3/">CRF Layer on the Top of BiLSTM - 3</a> CRF Loss Function</li>
<li><a href="https://createmomo.github.io/2017/10/17/CRF-Layer-on-the-Top-of-BiLSTM-4/">CRF Layer on the Top of BiLSTM - 4</a> Real Path Score</li>
<li><a href="https://createmomo.github.io/2017/11/11/CRF-Layer-on-the-Top-of-BiLSTM-5/">CRF Layer on the Top of BiLSTM - 5</a> The Total Score of All the Paths</li>
<li><a href="https://createmomo.github.io/2017/11/24/CRF-Layer-on-the-Top-of-BiLSTM-6/">CRF Layer on the Top of BiLSTM - 6</a> Infer the Labels for a New Sentence</li>
<li><a href="https://createmomo.github.io/2017/12/06/CRF-Layer-on-the-Top-of-BiLSTM-7/">CRF Layer on the Top of BiLSTM - 7</a> Chainer Implementation Warm Up</li>
<li><a href="https://createmomo.github.io/2017/12/07/CRF-Layer-on-the-Top-of-BiLSTM-8/">CRF Layer on the Top of BiLSTM - 8</a> Demo Code</li>
</ul>
<p><img src="/2019/07/18/Table-of-Contents/dog-mini.png" alt="The dog needs to find the best path to get his favorite bone toy and return home following the way he came"></p>
<hr>
<h4 id="My-Life"><a href="#My-Life" class="headerlink" title="My Life"></a><strong><em><a href="https://createmomo.github.io/2018/01/17/My-Life/">My Life</a></em></strong></h4>
      
    </div>
    <footer class="article-footer">
      <a data-url="http://createmomo.github.io/2019/07/18/Table-of-Contents/" data-id="ck0lc6z0n000cucp0tfofr6wf" class="article-share-link">Share</a>
      
        <a href="http://createmomo.github.io/2019/07/18/Table-of-Contents/#disqus_thread" class="article-comment-link">Comments</a>
      
      
    </footer>
  </div>
  
</article>


  
    <article id="post-Probabilistic-Graphical-Models-Revision-Notes" class="article article-type-post" itemscope itemprop="blogPost">
  <div class="article-meta">
    <a href="/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/" class="article-date">
  <time datetime="2019-01-07T00:00:00.000Z" itemprop="datePublished">2019-01-07</time>
</a>
    
  </div>
  <div class="article-inner">
    
    
      <header class="article-header">
        
  
    <h1 itemprop="name">
      <a class="article-title" href="/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/">Probabilistic Graphical Models Revision Notes</a>
    </h1>
  

      </header>
    
    <div class="article-entry" itemprop="articleBody">
      
        <h3 id="Last-Updated-2019-09-15"><a href="#Last-Updated-2019-09-15" class="headerlink" title="[Last Updated: 2019.09.15]"></a>[Last Updated: 2019.09.15]</h3><p>This note summarises the online course, <a href="https://www.coursera.org/specializations/probabilistic-graphical-models" target="_blank" rel="external">Probabilistic Graphical Models Specialization</a> on Coursera.<br><strong><strong>Any comments and suggestions are most welcome!</strong></strong><br></p>
        
          <p class="article-more-link">
            <a href="/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#more">Read More</a>
          </p>
        
      
    </div>
    <footer class="article-footer">
      <a data-url="http://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/" data-id="ck0lc6z0k000bucp04gjrw01p" class="article-share-link">Share</a>
      
        <a href="http://createmomo.github.io/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/#disqus_thread" class="article-comment-link">Comments</a>
      
      
    </footer>
  </div>
  
</article>


  
    <article id="post-Super-Machine-Learning-Revision-Notes" class="article article-type-post" itemscope itemprop="blogPost">
  <div class="article-meta">
    <a href="/2018/01/23/Super-Machine-Learning-Revision-Notes/" class="article-date">
  <time datetime="2018-01-23T00:00:00.000Z" itemprop="datePublished">2018-01-23</time>
</a>
    
  </div>
  <div class="article-inner">
    
    
      <header class="article-header">
        
  
    <h1 itemprop="name">
      <a class="article-title" href="/2018/01/23/Super-Machine-Learning-Revision-Notes/">Super Machine Learning Revision Notes</a>
    </h1>
  

      </header>
    
    <div class="article-entry" itemprop="articleBody">
      
        <h3 id="Last-Updated-06-01-2019"><a href="#Last-Updated-06-01-2019" class="headerlink" title="[Last Updated: 06/01/2019]"></a>[Last Updated: 06/01/2019]</h3><p>This article aims to summarise:</p>
<ul>
<li><strong>basic concepts</strong> in machine learning (e.g. gradient descent, back propagation etc.)</li>
<li><strong>different algorithms and various popular models</strong></li>
<li>some <strong>practical tips</strong> and <strong>examples</strong> were learned from my own practice and some online courses such as <a href="https://www.deeplearning.ai/" target="_blank" rel="external">Deep Learning AI</a>.</li>
</ul>
<p><strong>If you a student</strong> who is studying machine learning, hope this article could help you to shorten your revision time and bring you useful inspiration. <strong>If you are not a student</strong>, hope this article would be helpful when you cannot recall some models or algorithms.</p>
<p>Moreover, you can also treat it as a <strong>“Quick Check Guide”</strong>. Please be free to use Ctrl+F to search any key words interested you.</p>
<p><strong><strong>Any comments and suggestions are most welcome!</strong></strong><br></p>
        
          <p class="article-more-link">
            <a href="/2018/01/23/Super-Machine-Learning-Revision-Notes/#more">Read More</a>
          </p>
        
      
    </div>
    <footer class="article-footer">
      <a data-url="http://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/" data-id="ck0lc6z1a000ducp00zcyeavn" class="article-share-link">Share</a>
      
        <a href="http://createmomo.github.io/2018/01/23/Super-Machine-Learning-Revision-Notes/#disqus_thread" class="article-comment-link">Comments</a>
      
      
    </footer>
  </div>
  
</article>


  
    <article id="post-My-Life" class="article article-type-post" itemscope itemprop="blogPost">
  <div class="article-meta">
    <a href="/2018/01/17/My-Life/" class="article-date">
  <time datetime="2018-01-17T00:07:00.000Z" itemprop="datePublished">2018-01-17</time>
</a>
    
  </div>
  <div class="article-inner">
    
    
      <header class="article-header">
        
  
    <h1 itemprop="name">
      <a class="article-title" href="/2018/01/17/My-Life/">My Life</a>
    </h1>
  

      </header>
    
    <div class="article-entry" itemprop="articleBody">
      
        <hr>
<ul>
<li><strong>Reading for Learning English</strong><ul>
<li><em>[2017.07.25 -]</em> <strong><a href="https://www.amazon.com/Philip-C-Kolin-Successful-Writing/dp/B005E01154" target="_blank" rel="external">Successful Writing at Work, Ninth Edition</a></strong></li>
<li><em>[2017.09.27 -]</em> <strong><a href="https://geronimostilton.com/US-en/libri_top/scheda.php?id=747" target="_blank" rel="external">The Curse of the Cheese Pyramid</a> (Geronimo Stilton)</strong></li>
<li><em>[2016.12.17 - 2017.09.26]</em> <strong><a href="https://geronimostilton.com/US-en/libri_top/scheda.php?id=746" target="_blank" rel="external">Lost Treasure of the Emerald Eye</a> (Geronimo Stilton)</strong></li>
</ul>
</li>
<li><strong>Books</strong><ul>
<li><em>[2016.05.01 -]</em> <strong><a href="https://en.wikipedia.org/wiki/Love_in_the_Time_of_Cholera" target="_blank" rel="external">Love in the Time of Cholera</a></strong></li>
<li><em>[2015.02.24 - 2016.12.24]</em> <strong><a href="https://en.wikipedia.org/wiki/1Q84" target="_blank" rel="external">1Q84</a></strong></li>
<li><em>[2013.12.07 - 2015.04.10]</em> <strong><a href="https://www.amazon.com/gp/bookseries/B00YUQP6AE/ref=dp_st_0765382032" target="_blank" rel="external">The Three-Body Problem Series</a></strong></li>
</ul>
</li>
<li><strong>TV Series &amp; Movies</strong><ul>
<li><em>[2018.03.02 - ]</em> <strong><a href="https://en.wikipedia.org/wiki/The_Big_Bang_Theory" target="_blank" rel="external">The Big Bang Theory</a></strong></li>
<li><em>[2018.01.27 - 2018.03.01]</em> <strong><a href="http://www.bbc.co.uk/programmes/b018ttws/episodes/guide" target="_blank" rel="external">Sherlock</a></strong></li>
<li><em>[2016.04.13 - 2018.01.26]</em> <strong><a href="https://en.wikipedia.org/wiki/Downton_Abbey" target="_blank" rel="external">Downton Abbey</a></strong></li>
<li><em>[2015.04.01 - 2017.05.26]</em> <strong><a href="https://en.wikipedia.org/wiki/The_Vampire_Diaries" target="_blank" rel="external">the Vampire Diaries</a></strong></li>
</ul>
</li>
<li><strong>Piano</strong><ul>
<li><em>[2016.09.01 - 2017.02.05]</em> <strong>OST :</strong> <strong><a href="https://youtu.be/wHt6dfwe8rs?t=10s" target="_blank" rel="external">Just One Time is Enough</a> (<a href="https://en.wikipedia.org/wiki/Aska_Yang" target="_blank" rel="external">Aska Yang</a>)</strong></li>
</ul>
</li>
<li><strong>Sports</strong> (<em>First Time</em>)<ul>
<li><em>[2017]</em> <strong>Skiing</strong></li>
<li><em>[2014]</em> <strong>Ice Skating</strong></li>
<li><em>[2007]</em> <strong>Swimming</strong></li>
</ul>
</li>
</ul>

      
    </div>
    <footer class="article-footer">
      <a data-url="http://createmomo.github.io/2018/01/17/My-Life/" data-id="ck0lc6z020009ucp0g6o4vx6v" class="article-share-link">Share</a>
      
        <a href="http://createmomo.github.io/2018/01/17/My-Life/#disqus_thread" class="article-comment-link">Comments</a>
      
      
    </footer>
  </div>
  
</article>


  
    <article id="post-CRF-Layer-on-the-Top-of-BiLSTM-8" class="article article-type-post" itemscope itemprop="blogPost">
  <div class="article-meta">
    <a href="/2017/12/07/CRF-Layer-on-the-Top-of-BiLSTM-8/" class="article-date">
  <time datetime="2017-12-07T22:17:20.000Z" itemprop="datePublished">2017-12-07</time>
</a>
    
  </div>
  <div class="article-inner">
    
    
      <header class="article-header">
        
  
    <h1 itemprop="name">
      <a class="article-title" href="/2017/12/07/CRF-Layer-on-the-Top-of-BiLSTM-8/">CRF Layer on the Top of BiLSTM - 8</a>
    </h1>
  

      </header>
    
    <div class="article-entry" itemprop="articleBody">
      
        <h4 id="3-4-Demo"><a href="#3-4-Demo" class="headerlink" title="3.4 Demo"></a>3.4 Demo</h4><p>In this section, we will make two fake sentences which only have 2 words and 1 word respectively. Moreover, we will also randomly generate their true answers. Finally, we will show how to train the CRF Layer by using Chainer v2.0. All the codes including the CRF layer are avaialbe from <a href="https://github.com/createmomo/CRF-Layer-on-the-Top-of-BiLSTM" target="_blank" rel="external">GitHub</a>.<br></p>
        
          <p class="article-more-link">
            <a href="/2017/12/07/CRF-Layer-on-the-Top-of-BiLSTM-8/#more">Read More</a>
          </p>
        
      
    </div>
    <footer class="article-footer">
      <a data-url="http://createmomo.github.io/2017/12/07/CRF-Layer-on-the-Top-of-BiLSTM-8/" data-id="ck0lc6yzx0007ucp0ea68idze" class="article-share-link">Share</a>
      
        <a href="http://createmomo.github.io/2017/12/07/CRF-Layer-on-the-Top-of-BiLSTM-8/#disqus_thread" class="article-comment-link">Comments</a>
      
      
    </footer>
  </div>
  
</article>


  
    <article id="post-CRF-Layer-on-the-Top-of-BiLSTM-7" class="article article-type-post" itemscope itemprop="blogPost">
  <div class="article-meta">
    <a href="/2017/12/06/CRF-Layer-on-the-Top-of-BiLSTM-7/" class="article-date">
  <time datetime="2017-12-06T22:23:20.000Z" itemprop="datePublished">2017-12-06</time>
</a>
    
  </div>
  <div class="article-inner">
    
    
      <header class="article-header">
        
  
    <h1 itemprop="name">
      <a class="article-title" href="/2017/12/06/CRF-Layer-on-the-Top-of-BiLSTM-7/">CRF Layer on the Top of BiLSTM - 7</a>
    </h1>
  

      </header>
    
    <div class="article-entry" itemprop="articleBody">
      
        <h4 id="3-Chainer-Implementation"><a href="#3-Chainer-Implementation" class="headerlink" title="3 Chainer Implementation"></a>3 Chainer Implementation</h4><p>In this section, the structure of code will be explained. In addition, an important tip of implementing the CRF loss layer will also be given. Finally, the Chainer (version 2.0) implementation source code will be released in the next article.<br></p>
        
          <p class="article-more-link">
            <a href="/2017/12/06/CRF-Layer-on-the-Top-of-BiLSTM-7/#more">Read More</a>
          </p>
        
      
    </div>
    <footer class="article-footer">
      <a data-url="http://createmomo.github.io/2017/12/06/CRF-Layer-on-the-Top-of-BiLSTM-7/" data-id="ck0lc6yzl0005ucp0kpq3ti9i" class="article-share-link">Share</a>
      
        <a href="http://createmomo.github.io/2017/12/06/CRF-Layer-on-the-Top-of-BiLSTM-7/#disqus_thread" class="article-comment-link">Comments</a>
      
      
    </footer>
  </div>
  
</article>


  
    <article id="post-CRF-Layer-on-the-Top-of-BiLSTM-6" class="article article-type-post" itemscope itemprop="blogPost">
  <div class="article-meta">
    <a href="/2017/11/24/CRF-Layer-on-the-Top-of-BiLSTM-6/" class="article-date">
  <time datetime="2017-11-24T23:45:44.000Z" itemprop="datePublished">2017-11-24</time>
</a>
    
  </div>
  <div class="article-inner">
    
    
      <header class="article-header">
        
  
    <h1 itemprop="name">
      <a class="article-title" href="/2017/11/24/CRF-Layer-on-the-Top-of-BiLSTM-6/">CRF Layer on the Top of BiLSTM - 6</a>
    </h1>
  

      </header>
    
    <div class="article-entry" itemprop="articleBody">
      
        <h4 id="2-6-Infer-the-labels-for-a-new-sentence"><a href="#2-6-Infer-the-labels-for-a-new-sentence" class="headerlink" title="2.6 Infer the labels for a new sentence"></a>2.6 Infer the labels for a new sentence</h4><p>In the previous sections, we learned the structure of BiLSTM-CRF model and the details of CRF loss function. You can implement your own BiLSTM-CRF model by various opensource frameworks (Keras, Chainer, TensorFlow etc.). One of the greatest things is the backpropagation of on your model is automatically computed on these frameworks, therefore you do not need to implement the backpropagation by yourself to train your model (i.e. compute the gradients and to update parameters). Moreover, some frameworks have already implemented the CRF layer, so combining a CRF layer with your own model would be very easy by only adding about one line code.</p>
<p>In this section, we will explore how to infer the labels for a sentence during the test when our model is ready.<br></p>
        
          <p class="article-more-link">
            <a href="/2017/11/24/CRF-Layer-on-the-Top-of-BiLSTM-6/#more">Read More</a>
          </p>
        
      
    </div>
    <footer class="article-footer">
      <a data-url="http://createmomo.github.io/2017/11/24/CRF-Layer-on-the-Top-of-BiLSTM-6/" data-id="ck0lc6yzq0006ucp0gxilcdx4" class="article-share-link">Share</a>
      
        <a href="http://createmomo.github.io/2017/11/24/CRF-Layer-on-the-Top-of-BiLSTM-6/#disqus_thread" class="article-comment-link">Comments</a>
      
      
    </footer>
  </div>
  
</article>


  
    <article id="post-CRF-Layer-on-the-Top-of-BiLSTM-5" class="article article-type-post" itemscope itemprop="blogPost">
  <div class="article-meta">
    <a href="/2017/11/11/CRF-Layer-on-the-Top-of-BiLSTM-5/" class="article-date">
  <time datetime="2017-11-11T00:19:39.000Z" itemprop="datePublished">2017-11-11</time>
</a>
    
  </div>
  <div class="article-inner">
    
    
      <header class="article-header">
        
  
    <h1 itemprop="name">
      <a class="article-title" href="/2017/11/11/CRF-Layer-on-the-Top-of-BiLSTM-5/">CRF Layer on the Top of BiLSTM - 5</a>
    </h1>
  

      </header>
    
    <div class="article-entry" itemprop="articleBody">
      
        <h4 id="2-5-The-total-score-of-all-the-paths"><a href="#2-5-The-total-score-of-all-the-paths" class="headerlink" title="2.5 The total score of all the paths"></a>2.5 The total score of all the paths</h4><p>In the last section, we learned how to calculate the label path score of one path that is $e^{S_i}$. So far, we have one more problem which is needed to be solved, how to obtain the total score of all the paths ($ P_{total} = P_1 + P_2 + … + P_N = e^{S_1} + e^{S_2} + … + e^{S_N} $).</p>
<p>The simplest way to measure the total score is that: enumerating all the possible paths and sum their scores. Yes, you can calculate the total score in that way. However, it is very inefficient. The training time will be unbearable.<br></p>
        
          <p class="article-more-link">
            <a href="/2017/11/11/CRF-Layer-on-the-Top-of-BiLSTM-5/#more">Read More</a>
          </p>
        
      
    </div>
    <footer class="article-footer">
      <a data-url="http://createmomo.github.io/2017/11/11/CRF-Layer-on-the-Top-of-BiLSTM-5/" data-id="ck0lc6yzj0004ucp041en7hrx" class="article-share-link">Share</a>
      
        <a href="http://createmomo.github.io/2017/11/11/CRF-Layer-on-the-Top-of-BiLSTM-5/#disqus_thread" class="article-comment-link">Comments</a>
      
      
    </footer>
  </div>
  
</article>


  
    <article id="post-CRF-Layer-on-the-Top-of-BiLSTM-4" class="article article-type-post" itemscope itemprop="blogPost">
  <div class="article-meta">
    <a href="/2017/10/17/CRF-Layer-on-the-Top-of-BiLSTM-4/" class="article-date">
  <time datetime="2017-10-17T23:03:39.000Z" itemprop="datePublished">2017-10-17</time>
</a>
    
  </div>
  <div class="article-inner">
    
    
      <header class="article-header">
        
  
    <h1 itemprop="name">
      <a class="article-title" href="/2017/10/17/CRF-Layer-on-the-Top-of-BiLSTM-4/">CRF Layer on the Top of BiLSTM - 4</a>
    </h1>
  

      </header>
    
    <div class="article-entry" itemprop="articleBody">
      
        <h4 id="2-4-Real-path-score"><a href="#2-4-Real-path-score" class="headerlink" title="2.4 Real path score"></a>2.4 Real path score</h4><p>In section 2.3, we supposed that every possible path has a score $ P_{i} $ and there are totally $ N $ possible paths, the total score of all the paths is $ P_{total} = P_1 + P_2 + … + P_N = e^{S_1} + e^{S_2} + … + e^{S_N} $, $ e $ is the mathematical constant $ e $. </p>
<p>Obviously, there must be a path is the real one among all the possible paths. For exmaple, the real path of the sentence in section 1.2 is <strong>“START B-Person I-Person O B-Organization O END”</strong>. The others are incorrect such as “START B-Person B-Organization O I-Person I-Person B-Person”. $ e^{S_i} $ is the score of $ i^{th} $ path.</p>
<p>During the training process, the crf loss function only need two scores: the score of the real path and the total score of all the possbile paths. <strong>The proportion of the real path score among the scores of all the possible paths will be increased gradually.</strong></p>
<p>The calculation of a real path score, $e^{S_i}$, is very straightforward. </p>
        
          <p class="article-more-link">
            <a href="/2017/10/17/CRF-Layer-on-the-Top-of-BiLSTM-4/#more">Read More</a>
          </p>
        
      
    </div>
    <footer class="article-footer">
      <a data-url="http://createmomo.github.io/2017/10/17/CRF-Layer-on-the-Top-of-BiLSTM-4/" data-id="ck0lc6yz40002ucp03u60hg7a" class="article-share-link">Share</a>
      
        <a href="http://createmomo.github.io/2017/10/17/CRF-Layer-on-the-Top-of-BiLSTM-4/#disqus_thread" class="article-comment-link">Comments</a>
      
      
    </footer>
  </div>
  
</article>


  
    <article id="post-CRF-Layer-on-the-Top-of-BiLSTM-3" class="article article-type-post" itemscope itemprop="blogPost">
  <div class="article-meta">
    <a href="/2017/10/08/CRF-Layer-on-the-Top-of-BiLSTM-3/" class="article-date">
  <time datetime="2017-10-08T00:05:31.000Z" itemprop="datePublished">2017-10-08</time>
</a>
    
  </div>
  <div class="article-inner">
    
    
      <header class="article-header">
        
  
    <h1 itemprop="name">
      <a class="article-title" href="/2017/10/08/CRF-Layer-on-the-Top-of-BiLSTM-3/">CRF Layer on the Top of BiLSTM - 3</a>
    </h1>
  

      </header>
    
    <div class="article-entry" itemprop="articleBody">
      
        <h4 id="2-3-CRF-loss-function"><a href="#2-3-CRF-loss-function" class="headerlink" title="2.3 CRF loss function"></a>2.3 CRF loss function</h4><p>The CRF loss function is consist of the real path score and the total score of all the possible paths. The real path should have the highest score among those of all the possible paths.</p>
<p>For example, if we have these labels in our dataset as shown in the table:</p>
<div class="table-container">
<table>
<thead>
<tr>
<th style="text-align:left">Label</th>
<th style="text-align:right">Index</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left">B-Person</td>
<td style="text-align:right">0</td>
</tr>
<tr>
<td style="text-align:left">I-Person</td>
<td style="text-align:right">1</td>
</tr>
<tr>
<td style="text-align:left">B-Organization</td>
<td style="text-align:right">2</td>
</tr>
<tr>
<td style="text-align:left">I-Organization</td>
<td style="text-align:right">3</td>
</tr>
<tr>
<td style="text-align:left">O</td>
<td style="text-align:right">4</td>
</tr>
<tr>
<td style="text-align:left">START</td>
<td style="text-align:right">5</td>
</tr>
<tr>
<td style="text-align:left">END</td>
<td style="text-align:right">6</td>
</tr>
</tbody>
</table>
</div>
<p>We also have a sentence which has 5 words. The possible paths could be:</p>
<ul>
<li>1) START B-Person B-Person B-Person B-Person B-Person END</li>
<li>2) START B-Person I-Person B-Person B-Person B-Person END</li>
<li>…</li>
<li><strong>10) START B-Person I-Person O B-Organization O END</strong></li>
<li>…</li>
<li>N) O O O O O O O</li></ul>
        
          <p class="article-more-link">
            <a href="/2017/10/08/CRF-Layer-on-the-Top-of-BiLSTM-3/#more">Read More</a>
          </p>
        
      
    </div>
    <footer class="article-footer">
      <a data-url="http://createmomo.github.io/2017/10/08/CRF-Layer-on-the-Top-of-BiLSTM-3/" data-id="ck0lc6yza0003ucp07dmsiv9l" class="article-share-link">Share</a>
      
        <a href="http://createmomo.github.io/2017/10/08/CRF-Layer-on-the-Top-of-BiLSTM-3/#disqus_thread" class="article-comment-link">Comments</a>
      
      
    </footer>
  </div>
  
</article>


  


  <nav id="page-nav">
    <span class="page-number current">1</span><a class="page-number" href="/page/2/">2</a><a class="extend next" rel="next" href="/page/2/">__('next') &raquo;</a>
  </nav>
</section>
        
          <aside id="sidebar">
  
    

  
    

  
    
  
    
  <div class="widget-wrap">
    <h3 class="widget-title">Archives</h3>
    <div class="widget">
      <ul class="archive-list"><li class="archive-list-item"><a class="archive-list-link" href="/archives/2019/07/">July 2019</a></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2019/01/">January 2019</a></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2018/01/">January 2018</a></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2017/12/">December 2017</a></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2017/11/">November 2017</a></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2017/10/">October 2017</a></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2017/09/">September 2017</a></li></ul>
    </div>
  </div>


  
    
  <div class="widget-wrap">
    <h3 class="widget-title">Recent Posts</h3>
    <div class="widget">
      <ul>
        
          <li>
            <a href="/2019/07/18/Table-of-Contents/">Table of Contents</a>
          </li>
        
          <li>
            <a href="/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/">Probabilistic Graphical Models Revision Notes</a>
          </li>
        
          <li>
            <a href="/2018/01/23/Super-Machine-Learning-Revision-Notes/">Super Machine Learning Revision Notes</a>
          </li>
        
          <li>
            <a href="/2018/01/17/My-Life/">My Life</a>
          </li>
        
          <li>
            <a href="/2017/12/07/CRF-Layer-on-the-Top-of-BiLSTM-8/">CRF Layer on the Top of BiLSTM - 8</a>
          </li>
        
      </ul>
    </div>
  </div>

  
</aside>
        
      </div>
      <footer id="footer">
  
  <div class="outer">
    <div id="footer-info" class="inner">
      &copy; 2019 CreateMoMo<br>
      Powered by <a href="http://hexo.io/" target="_blank">Hexo</a>
    </div>
  </div>
</footer>
    </div>
    <nav id="mobile-nav">
  
    <a href="/" class="mobile-nav-link">Home</a>
  
    <a href="/archives" class="mobile-nav-link">Archives</a>
  
</nav>
    
<script>
  var disqus_shortname = 'createmomo';
  
  (function(){
    var dsq = document.createElement('script');
    dsq.type = 'text/javascript';
    dsq.async = true;
    dsq.src = '//' + disqus_shortname + '.disqus.com/count.js';
    (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
  })();
</script>


<script src="//ajax.googleapis.com/ajax/libs/jquery/2.0.3/jquery.min.js"></script>


  <link rel="stylesheet" href="/fancybox/jquery.fancybox.css">
  <script src="/fancybox/jquery.fancybox.pack.js"></script>


<script src="/js/script.js"></script>

  </div>
<script type="text/x-mathjax-config">
    MathJax.Hub.Config({
        tex2jax: {
            inlineMath: [ ["$","$"], ["\\(","\\)"] ],
            skipTags: ['script', 'noscript', 'style', 'textarea', 'pre', 'code'],
            processEscapes: true
        }
    });
    MathJax.Hub.Queue(function() {
        var all = MathJax.Hub.getAllJax();
        for (var i = 0; i < all.length; ++i)
            all[i].SourceElement().parentNode.className += ' has-jax';
    });
</script>
<!-- <script src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>-->
<script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-MML-AM_CHTML"></script><!-- hexo-inject:begin --><!-- hexo-inject:end -->
</body>
</html>